Face recognition system, face recognition method, display control apparatus, display control method, and display control program

ABSTRACT

There is provided a display control apparatus that allows an operator to grasp a factor leading a face recognition result at a glance and to confirm or modify the face recognition result on the spot. The display control apparatus comprises a similarity acquirer that acquires a similarity between each pair of partial regions of face images by performing collation processing between the each pair of partial regions of the face images, and a display controller that controls to overlay, on the face images, at least one of a first region the similarity of which exceeds a threshold and a second region the similarity of which does not exceed the threshold, and display the overlaid face images. The display controller controls to overlay and display the first region and the second region in contrast with each other on the face images.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation Application of U.S.application Ser. No. 16/129,275 filed on Sep. 12, 2018, which is aContinuation Application of U.S. application Ser. No. 15/758,185 filedon Mar. 7, 2018, which issued as U.S. Pat. No. 10,671,837, which is aNational Stage of International Application No. PCT/JP2016/066778 filedon Jun. 6, 2016, and which claims priority based on Japanese PatentApplication No. 2015-176534 filed on Sep. 8, 2015, the contents of allof which are incorporated herein by reference in their entirety.

TECHNICAL FIELD

The present invention relates to a face recognition system, a facerecognition method, a display control apparatus, a display controlmethod, and a display control program.

BACKGROUND ART

In the above technical field, patent literature 1 discloses a techniqueof searching for an image from an image database by collation for eachpair of portions (eyes, mouths, or noses) of face images. Patentliterature 2 discloses a technique of notifying a user of acorresponding response sentence based on a similarity obtained bycollation for each pair of portion regions (eye regions, mouth regions,or nose regions) of face images.

CITATION LIST Patent Literature

Patent literature 1: Japanese Patent Laid-Open No. 2010-146581

Patent literature 2: Japanese Patent Laid-Open No. 2005-078413

SUMMARY OF THE INVENTION Technical Problem

In the techniques described in the above patent literatures, however,only information based on a face recognition result is output.Therefore, an operator cannot grasp a factor leading the facerecognition result at a glance to confirm or modify the face recognitionresult on the spot.

The present invention enables to provide a technique of solving theabove-described problem.

Solution to Problem

One example aspect of the present invention provides a display controlapparatus comprising:

a similarity acquirer that acquires a similarity between each pair ofpartial regions of face images by performing collation processingbetween the each pair of partial regions of the face images; and

a display controller that controls to overlay, on the face images, atleast one of a first region the similarity of which exceeds a thresholdand a second region the similarity of which does not exceed thethreshold, and display the overlaid face images.

Another example aspect of the present invention provides a displaycontrol method comprising:

acquiring a similarity between each pair of partial regions of faceimages by performing collation processing between the each pair ofpartial regions of the face images; and

controlling to overlay, on the face images, at least one of a firstregion the similarity of which exceeds a threshold and a second regionthe similarity of which does not exceed the threshold, and display theoverlaid face images.

Still other example aspect of the present invention provides a displaycontrol program for causing a computer to execute a method, comprising:

acquiring a similarity between each pair of partial regions of faceimages by performing collation processing between the each pair ofpartial regions of the face images; and

controlling to overlay, on the face images, at least one of a firstregion the similarity of which exceeds a threshold and a second regionthe similarity of which does not exceed the threshold, and display theoverlaid face images.

Still other example aspect of the present invention provides a facerecognition system comprising:

an image capturer that generates a first face image;

an accumulator that accumulates a second face image;

a deriving unit that derives a similarity between each pair of a partialregion of the first face image and a partial region of the second faceimage by performing collation processing between the each pair ofpartial regions of the first face image and the second face image;

-   -   a face recognizer that recognizes the first face image based on        derived similarities of the partial regions; and

a display unit that overlays, on at least one of the first face imageand the second face image, a result of recognizing the first face imageand at least one of a first region the similarity of which exceeds athreshold and a second region the similarity of which does not exceedthe threshold, and displays the overlaid face images.

Still other example aspect of the present invention provides a facerecognition method comprising:

deriving a similarity between each pair of a partial region of a firstface image generated by an image capturer and a partial region of asecond face image accumulated by an accumulator by performing collationprocessing between the each pair of partial regions of the first faceimage and the second face image;

recognizing the first face image based on derived similarities of thepartial regions; and

overlaying, on at least one of the first face image and the second faceimage, a result of recognizing the first face image and at least one ofa first region the similarity of which exceeds a threshold and a secondregion the similarity of which does not exceed the threshold, anddisplaying the overlaid face images on a display unit.

Advantageous Effects of Invention

According to the present invention, since an operator can grasp a factorleading a face recognition result at a glance, he/she can confirm ormodify the face recognition result on the spot.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing the arrangement of a display controlapparatus according to the first example embodiment of the presentinvention;

FIG. 2A is a view showing an outline of a display screen according tothe second example embodiment of the present invention;

FIG. 2B is a view showing another outline of the display screenaccording to the second example embodiment of the present invention;

FIG. 3 is a block diagram showing the arrangement of a face recognitionsystem according to the second example embodiment of the presentinvention;

FIG. 4 is a sequence chart showing the operation procedure of the facerecognition system according to the second example embodiment of thepresent invention;

FIG. 5 is a block diagram showing the functional arrangement of adisplay control apparatus according to the second example embodiment ofthe present invention;

FIG. 6 is a table showing the structure of a face image storageaccording to the second example embodiment of the present invention;

FIG. 7 is a table showing the structure of a threshold storage accordingto the second example embodiment of the present invention;

FIG. 8A is a table showing the structure of a mark storage according tothe second example embodiment of the present invention;

FIG. 8B is a table showing the structure of an overlaying image datageneration table according to the second example embodiment of thepresent invention;

FIG. 9 is a table showing the structure of an overlaid display datageneration table according to the second example embodiment of thepresent invention;

FIG. 10 is a block diagram showing the functional arrangement of a facerecognition apparatus according to the second example embodiment of thepresent invention;

FIG. 11A is a table showing the structure of a captured face imagedatabase according to the second example embodiment of the presentinvention;

FIG. 11B is a table showing the structure of a collation target faceimage database according to the second example embodiment of the presentinvention;

FIG. 12 is a table showing the structure of a partial region databaseaccording to the second example embodiment of the present invention;

FIG. 13 is a table showing the structure of a face recognition tableaccording to the second example embodiment of the present invention;

FIG. 14 is a block diagram showing the functional arrangement of anoperator terminal according to the second example embodiment of thepresent invention;

FIG. 15 is a table showing the structure of an operation processingtable according to the second example embodiment of the presentinvention;

FIG. 16 is a block diagram showing the hardware arrangement of thedisplay control apparatus according to the second example embodiment ofthe present invention;

FIG. 17 is a flowchart illustrating the processing procedure of thedisplay control apparatus according to the second example embodiment ofthe present invention;

FIG. 18A is a flowchart illustrating an example of the procedure of markgeneration processing according to the second example embodiment of thepresent invention;

FIG. 18B is a flowchart illustrating another example of the procedure ofthe mark generation processing according to the second exampleembodiment of the present invention;

FIG. 19 is a block diagram showing the hardware arrangement of the facerecognition apparatus according to the second example embodiment of thepresent invention;

FIG. 20 is a flowchart illustrating the processing procedure of the facerecognition apparatus according to the second example embodiment of thepresent invention;

FIG. 21 is a view showing an outline of a display screen according tothe third example embodiment of the present invention;

FIG. 22 is a block diagram showing the functional arrangement of adisplay control apparatus according to the third example embodiment ofthe present invention;

FIG. 23 is a table showing the structure of a region connection tableaccording to the third example embodiment of the present invention;

FIG. 24 is a flowchart illustrating the procedure of the regionconnection processing of the display control apparatus according to thethird example embodiment of the present invention;

FIG. 25 is a view showing an outline of a display screen according tothe fourth example embodiment of the present invention;

FIG. 26 is a sequence chart showing the operation procedure of a facerecognition system according to the fourth example embodiment of thepresent invention;

FIG. 27 is a block diagram showing the functional arrangement of adisplay control apparatus according to the fourth example embodiment ofthe present invention;

FIG. 28 is a flowchart illustrating the procedure of the display controlprocessing of the display control apparatus according to the fourthexample embodiment of the present invention;

FIG. 29 is a view showing an outline of a display screen according tothe fifth example embodiment of the present invention;

FIG. 30 is a sequence chart showing the operation procedure of a facerecognition system according to the fifth example embodiment of thepresent invention;

FIG. 31 is a block diagram showing the functional arrangement of adisplay control apparatus according to the fifth example embodiment ofthe present invention;

FIG. 32 is a flowchart illustrating the procedure of the display controlprocessing of the display control apparatus according to the fifthexample embodiment of the present invention;

FIG. 33 is a view showing an outline of a display screen according tothe sixth example embodiment of the present invention;

FIG. 34 is a sequence chart showing the operation procedure of a facerecognition system according to the sixth example embodiment of thepresent invention;

FIG. 35 is a block diagram showing the functional arrangement of adisplay control apparatus according to the sixth example embodiment ofthe present invention;

FIG. 36 is a flowchart illustrating the procedure of the display controlprocessing of the display control apparatus according to the sixthexample embodiment of the present invention;

FIG. 37 is a block diagram showing the functional arrangement of a facerecognition apparatus according to the sixth example embodiment of thepresent invention;

FIG. 38 is a table showing the structure of a face recognition history(learning) database according to the sixth example embodiment of thepresent invention;

FIG. 39 is a table showing the structure of a region-of-interest andregion-of-non-interest generation table according to the sixth exampleembodiment of the present invention;

FIG. 40 is a flowchart illustrating the procedure of region-of-interestand/or region-of-non-interest generation processing according to thesixth example embodiment of the present invention;

FIG. 41 is a block diagram showing the arrangement of a face recognitionsystem according to the seventh example embodiment of the presentinvention;

FIG. 42A is a block diagram showing the arrangement of a facerecognition system according to the eighth example embodiment of thepresent invention;

FIG. 42B is a block diagram showing another arrangement of the facerecognition system according to the eighth example embodiment of thepresent invention; and

FIG. 43 is a block diagram showing the functional arrangement of acommunication terminal as a face recognition system according to theninth example embodiment of the present invention.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Example embodiments of the present invention will now be described indetail with reference to the drawings. It should be noted that therelative arrangement of the components, the numerical expressions andnumerical values set forth in these example embodiments do not limit thescope of the present invention unless it is specifically statedotherwise.

First Example Embodiment

A display control apparatus 100 according to the first exampleembodiment of the present invention will be described with reference toFIG. 1. The display control apparatus 100 is an apparatus that controlsdisplay of a face image to undergo face recognition.

As shown in FIG. 1, the display control apparatus 100 includes asimilarity acquirer 101 and a display controller 102. The similarityacquirer 101 acquires a similarity between each pair of partial regionsof face images by performing collation processing between the each pairof partial regions of the face images. The display controller 102controls to overlay, on the face images, at least one of the firstregion the similarity of which exceeds a threshold and the second regionthe similarity of which does not exceed the threshold, and display theoverlaid face images.

According to this example embodiment, when the degree of similarity foreach pair of partial regions of face images is overlaid and displayed onthe face images, the operator can grasp a factor leading a facerecognition result at a glance, and thus can confirm or modify the facerecognition result on the spot.

Second Example Embodiment

A face recognition system including a display control apparatusaccording to the second example embodiment of the present invention willbe described next. The display control apparatus according to thisexample embodiment acquires a similarity derived by performing collationfor each pair of partial regions of face images in a face recognitionapparatus, and controls to overlay and display, on the face images,marks each indicating whether the acquired similarity exceeds athreshold or not. This allows the user or operator to grasp a factor fora face recognition result at a glance, and confirm or modify the facerecognition result on the spot.

<<Display Screen of Face Recognition Result>>

FIG. 2A is a view showing an outline of the display screen of a terminal210 according to this example embodiment. Note that the display screenshown in FIG. 2A is not limited, and may be the display screen of amobile terminal, the display screen of a PC (Personal Computer), or thedisplay screen of an apparatus dedicated for face recognition orsurveillance.

In the display screen of the terminal 210 shown on the left side of FIG.2A, a mark “◯” indicating that the similarity of each pair of partialregions (in FIG. 2A, 5 (vertical)×5 (horizontal)=25) exceeds apredetermined threshold as a result of comparison is overlaid on acaptured face image or recognition target face image 211. The size ofthe mark “◯” corresponds to a difference value obtained by subtractingthe predetermined threshold from the similarity. That is, a mark “◯” 212of a large size indicates a partial region whose similarity is muchlarger than the predetermined threshold. On the other hand, a mark “◯”213 of a small size indicates a partial region whose similarity is notmuch larger than the predetermined threshold. In this way, the marks “◯”are overlaid and displayed so as to discriminate between the high andlow similarities. Note that in the display screen on the left side, amark “●” indicates a partial region whose similarity does not exceed thepredetermined threshold. From the display screen on the left side, theuser or operator can understand at a glance that the peripheral regionsof eyes and the central region of a mouth are similar regions.

In the display screen of the terminal 210 shown on the right side ofFIG. 2A, a mark “□” indicating that the similarity of each pair of thepartial regions (in FIG. 2A, 5 (vertical)×5 (horizontal)=25) exceeds apredetermined threshold as a result of comparison is overlaid on thecaptured face image or recognition target face image 211. The size ofthe mark “□” corresponds to a difference value obtained by subtractingthe predetermined threshold from the similarity. That is, a mark “□” 214of a large size indicates a partial region whose similarity is muchlarger than the predetermined threshold. On the other hand, a mark “□”215 of a small size indicates a partial region whose similarity is notmuch larger than the predetermined threshold. Note that in the displayscreen on the right side, a mark “●” indicates a partial region whosesimilarity does not exceed the predetermined threshold. Similarly to thedisplay screen on the left side, from the display screen on the rightside, the user or operator can understand at a glance that theperipheral regions of the eyes and the central region of the mouth aresimilar regions.

Note that FIG. 2A shows the marks “O” of large and small sizes and themarks “□” of large and small sizes are shown as marks each indicatingthe similarity. The shape of the mark is not limited to them shown inFIG. 2A. The degree of similarity corresponds to the size of the markbut may be represented by the type or chromaticity of the color of themark, the density of the mark, or the transparency when the mark isoverlaid on the face images.

FIG. 2B is a view showing another outline of the display screen of theterminal 210 according to this example embodiment.

A captured face image 221 and an accumulated recognition target faceimage 222 are displayed in the display screen of the terminal 210 shownon the left side of FIG. 2B. In the accumulated recognition target faceimage 222, a similar portion 223 of 5 (vertical)×4 (horizontal)=20partial regions is filled with a predetermined transparent color(represented by backward diagonal hatching in FIG. 2B). On the otherhand, a dissimilar portion 224 is not filled. The similar portionindicates a region where the partial region similarity exceeds thepredetermined threshold. Note that this overlay and display processingis implemented by overlaying, as similar region marks, frames filledwith the transparent color on the partial regions of the face image 222.

A captured face image 231 and an accumulated recognition target faceimage 232 are displayed in the display screen of the terminal 210 shownon the right side of FIG. 2B. In the accumulated recognition target faceimage 232, a dissimilar portion 233 of 5 (vertical)×4 (horizontal)=20partial regions is filled with a predetermined transparent color(represented by forward diagonal hatching in FIG. 2B). On the otherhand, a similar portion 234 is not filled. The dissimilar portionindicates a region where the partial region similarity does not exceedthe predetermined threshold. Note that this overlay and displayprocessing is implemented by overlaying, as dissimilar region marks,frames filled with the transparent color on the partial regions of theface image 232.

Note that FIG. 2B shows an example in which the similar region marks ordissimilar region marks are overlaid on the accumulated recognitiontarget face image 222 or 232. However, the similar region marks ordissimilar region marks may be overlaid on the captured face image 221or 231. Alternatively, one or both of a similar region mark and adissimilar region mark may be overlaid on the face images. If both themarks are overlaid, the similar region mark and the dissimilar regionmark are displayed to be collated. A face image overlaid and displayedon the screen may be one of the captured face image 221 or 231 and theaccumulated recognition target face image 222 or 232.

In FIGS. 2A and 2B, the captured face image is compared with theaccumulated recognition target face image to calculate the similarity.However, a plurality of captured face images or a plurality ofaccumulated face images may be compared with each other. The markincludes a symbol of a predetermined shape, a colored partial region, ora filled partial region.

<<Face Recognition System>>

The arrangement and operation of a face recognition system 300 accordingto this example embodiment will be described with reference to FIGS. 3and 4.

(System Arrangement)

FIG. 3 is a block diagram showing the arrangement of the facerecognition system 300 according to this example embodiment.

The face recognition system 300 includes image capturing apparatuses(cameras) 320 to 32 m, terminals 210 to 21 n, a display controlapparatus 330, and a face recognition apparatus 340, all of which areconnected by a network 350.

The image capturing apparatuses (cameras) 320 to 32 m includesurveillance cameras arranged in an airport and the like. The terminals210 to 21 n may include the operator terminal 210 for surveillancearranged in an airport or the like, and the note PC 21 i or a mobileterminal (smartphone, tablet, or the like) as a user terminal. Thedisplay control apparatus 330 compares, with a threshold, a similarityas a result of comparison for each pair of partial regions of faceimages in the face recognition apparatus 340, and controls display onthe terminals 210 to 21 n by overlaying the result on the face images.The face recognition apparatus 340 extracts a face image from an imagecaptured by one of the image capturing apparatuses (cameras) 320 to 32m, derives a similarity by comparing each partial region of theextracted face image with a corresponding partial region of a face imageread out from an accumulator, and recognizes a face based on thesimilarities. The face recognition apparatus 340 transmits the facerecognition result and the similarities of the pairs of partial regionsto the display control apparatus 330, and also transmits the facerecognition result to the terminals 210 to 21 n.

Note that the display control apparatus 330 and the face recognitionapparatus 340 may be connected by a local network, as indicated by adashed line.

(Operation Sequence)

FIG. 4 is a sequence chart showing the operation procedure of the facerecognition system 300 according to this example embodiment. Note thatin FIG. 4, the image capturing apparatuses (cameras) 320 to 32 m arerepresented by the camera 320, and the terminals 210 to 21 n arerepresented by the operator terminal 210.

In step S401, the face recognition system 300 is initialized. In stepS403, the camera 320 captures a person and transmits a captured image tothe face recognition apparatus 340. Note that the camera 320 may capturea crowd in the lobby of an airport or on a road instead of a specificperson, and the face recognition apparatus 340 or the like extracts eachperson, and extracts a face image of each person. An arrangement inwhich an advanced image processor is provided in the camera 320 and onlythe feature of a face image is transmitted to the face recognitionapparatus 340 may be adopted.

Upon receiving the captured image from the camera 320, the facerecognition apparatus 340 divides the face image into partial regions instep S405. Note that the partial regions may be rectangular regionshaving different areas in accordance with the importance of facerecognition or regions each surrounded by a curve, instead of regionsobtained by dividing the image in a grid pattern shown in FIG. 2A or 2B.Furthermore, the partial regions may be regions respectively includingportions such as eyes, a nose, and a mouth extracted from the capturedface image but are not limited to them.

In step S407, the face recognition apparatus 340 calculates a similarityby comparing corresponding partial regions of the captured face imageand each face image read out from a face image database in which faceimages are accumulated. Note that enlargement/reduction processing forassociating the partial regions of the face images is also performed.However, the aspect ratio of the face in each of the face imagescaptured at the same angle desirably remains the same. To calculate thesimilarity between the partial regions, for example, the accumulation ofcomparison results of luminances, colors, or the like on a pixel basis,the accumulation of the features of regions obtained by subdividing eachpartial region, or the feature of the luminance or color of each entirepartial region or the feature of an edge in each partial region is used.The present invention, however, is not limited to them. For example, anSSD (Sum of Squared Difference), an SAD (Sum of Absolute Difference), anormalized correlation value, or the like may be calculated as thesimilarity using the arrangement of the luminance values of therespective pixels of each partial region or each of regions obtained bysubdividing each partial region.

In step S409, the face recognition apparatus 340 accumulates thesimilarity of each pair of partial regions in consideration of a weight,and comparing the accumulated values with a threshold, therebyperforming face recognition of determining similarity or dissimilarity.If there exists a similar face, the user is notified of the face as arecognition result. At the same time, in step S411, the face recognitionapparatus 340 transmits, to the display control apparatus 330, the facerecognition result and the collation result (similarity) of each pair ofpartial regions of the face images. Note that the operator terminal 210may be directly notified of the face recognition result.

In step S413, the display control apparatus 330 compares, with apredetermined threshold, each partial region similarity received fromthe face recognition apparatus 340. In step S415, the display controlapparatus 330 generates an overlaid display image data by overlaying apredetermined mark on the partial region of the face image whosesimilarity exceeds the predetermined threshold and/or the partial regionwhose similarity does not exceed the predetermined threshold. In stepS417, the display control apparatus 330 transmits the overlaid displayimage data to the operator terminal 210.

In step S419, the operator terminal 210 displays an overlaid imagereceived from the display control apparatus 330. Note that the operatorterminal 210 displays the face recognition result (similarity ordissimilarity) by the face recognition apparatus 340 at the same time.In step S421, the operator terminal 210 transmits, to the camera 320,the display control apparatus 330, and the face recognition apparatus340, an operation performed by the operator by viewing the overlay anddisplay.

Note that FIG. 4 shows a case in which the display control apparatus 330generates an overlaid display image. An image indicating the marktransmitted from the display control apparatus 330 may be overlaid onthe face images in the operator terminal 210.

<<Functional Arrangement of Display Control Apparatus>>

FIG. 5 is a block diagram showing the functional arrangement of thedisplay control apparatus 330 according to this example embodiment.

The display control apparatus 330 includes a communication controller501, a face image acquirer 502, a face image storage 503, a similarityacquirer 504, a comparator 505, and a threshold storage 506. The displaycontrol apparatus 330 further includes a mark generator 507, a markstorage 508, an image overlay unit 509, and an overlaid imagetransmitter 510.

The communication controller 501 controls communication with the facerecognition apparatus 340 or the operator terminal 210 via the network350. The face image acquirer 502 acquires a face image having undergoneface recognition from the face recognition apparatus 340 via thecommunication controller 501. The face image having undergone facerecognition includes at least one of a captured face image and anaccumulated recognition target face image.

The similarity acquirer 504 acquires each partial region similarity inassociation with a partial region ID (Identifier) from the facerecognition apparatus 340. The comparator 505 compares each partialregion similarity with the predetermined threshold, and outputs acomparison result (data indicating whether the similarity exceeds thethreshold). The threshold storage 506 stores the threshold to becompared with each partial region similarity in the comparator 505. Asthe threshold, one threshold used to determine whether the similarityexceeds the threshold, or two thresholds used to determine whether thesimilarity exceeds the first threshold or the second threshold (firstthreshold>one threshold>second threshold) may be stored. Furthermore,various thresholds can be set in accordance with information of whichthe operator is notified by an overlaid image.

The mark generator 507 generates a mark to be overlaid on each partialregion of the face image in accordance with the output from thecomparator 505. Note that a mark is generated by selecting, inaccordance with the output from the comparator 505, a mark to be usedfrom the mark storage 508. The mark storage 508 stores a mark to beoverlaid on each partial region of the face image in accordance with theoutput from the comparator 505. Note that the mark includes the fill ofthe partial region.

The image overlay unit 509 overlays, on the face images stored in theface image storage unit 503, the mark generated by the mark generator507 and indicating whether the similarity exceeds the threshold. Theoverlaid image transmitter 510 transmits, to the operator terminal 210via the network 350, the face image in which the marks are overlaid onthe partial regions by the image overlay unit 509.

Note that in FIG. 5, the comparator 505, the threshold storage 506, themark generator 507, the mark storage 508, the image overlay unit 509,and the overlaid image transmitter 510 correspond to the displaycontroller 102 shown in FIG. 1.

(Face Image Storage)

FIG. 6 is a table showing the structure of the face image storage 503according to this example embodiment. The face image storage 503 storesthe face image and the face recognition result in the face recognitionapparatus 340 for generating an overlaid image with the marks. Note thatthe structure of the face image storage 503 is not limited to that shownin FIG. 6.

The face image storage 503 stores a face recognition ID 601 foridentifying face recognition processing, a captured image ID 602 foridentifying an image captured by the camera 320, an accumulated image ID603 for identifying an accumulated image as a recognition target in theface recognition apparatus 340, and a face recognition result 604 in theface recognition apparatus 340. The face image storage 503 also storesan overlaying image flag 605 indicating whether a face image to undergooverlay is one or both of a captured image and an accumulated image.

(Threshold Storage)

FIG. 7 is a table showing the structure of the threshold storage 506according to this example embodiment. The threshold storage 506 stores athreshold that corresponds to an overlaying mark and is to be comparedwith the similarity of each pair of partial regions of the face images.Note that the structure of the threshold storage 506 is not limited tothat shown in FIG. 7.

The threshold storage 506 stores a threshold 702 in association with anapplication 701 of the threshold in comparison with the similarity. InFIG. 7, for example, one threshold Th0 for dividing the partial regionbased on the similarity, which is used to determine whether thesimilarity exceeds the threshold, and two thresholds Th1 (firstthreshold) and Th2 (second threshold) for separating the partial regionof a high similarity exceeding the first threshold and separating thepartial region of a low similarity not exceeding the second thresholdare stored. Note that a threshold for separating the similarity may befurther provided to finely change the overlaying mark.

(Mark Storage)

FIG. 8A is a table showing the structure of the mark storage 508according to this example embodiment. The mark storage 508 storesvarious marks to be selected in consideration of the condition of thecomparison result between each partial region similarity and thethreshold and the condition of the mark indicating the comparison resultto the operator. Note that the structure of the mark storage 508 is notlimited to that shown in FIG. 8.

The mark storage 508 stores, in association with a mark ID 811 foridentifying a mark, a shape 812 of the mark, basic image data 813 of themark, and a change 814 corresponding to the similarity. The basic imagedata 813 stores the size, color, and density as the basis of the mark,image data on a pixel basis, or the like.

(Overlaying Image Data Generation Table)

FIG. 8B is a table showing the structure of an overlaying image datageneration table 820 according to this example embodiment. Theoverlaying image data generation table 820 is used in processing up tothe mark generator 507 to generate overlaying image data of each partialregion.

In association with each partial region ID 822 of a display face imageID 821 for identifying a face image on which a mark is overlaid anddisplayed, the overlaying image data generation table 820 stores apartial region similarity 823, a comparison result 824 indicatingwhether or not the threshold is exceeded, a mark ID 825 for indicatingthe comparison result 824, and a level 826 corresponding to thecomparison result 824, for example, (similarity-threshold) or(threshold-similarity). The overlaying image data generation table 820stores overlaying image data 827 to be overlaid on each partial region.

(Overlaid Display Data Generation Table)

FIG. 9 is a table showing the structure of an overlaid display datageneration table 900 according to this example embodiment. The overlaiddisplay data generation table 900 is used in the processing of the imageoverlay unit 509 to generate display image data by overlaying theoverlaying image data of each partial region on each partial region ofthe face image.

The overlaid display data generation table 900 stores face image data902 in association with the display face image ID 821. The overlaiddisplay data generation table 900 stores the overlaying image data 827of each partial region generated by the mark generator 507 and overlaidresult data 904 of each partial region. In addition, the overlaiddisplay data generation table 900 stores display image data 905 obtainedby setting the overlaid result data 904 of the respective partialregions as the entire face image.

Note that in FIG. 9, an overlaying image is generated for each partialregion. However, overlaying image data may be generated by individuallycollecting the marks, and the display face image data and the overlayingimage data may be collectively overlaid.

<<Functional Arrangement of Face Recognition Apparatus>>

FIG. 10 is a block diagram showing the functional arrangement of theface recognition apparatus 340 according to this example embodiment.

The face recognition apparatus 340 includes a communication controller1001, a captured image acquirer 1002, a face image extractor 1003, acaptured face image database 1004, a collation target face imagedatabase 1005, and a partial region database 1006. The face recognitionapparatus 340 includes a partial region generator 1007, a partial regionsimilarity calculator 1008, a face recognizer 1009, a face recognitionresult and similarity transmitter 1010, and an operator's operationacquirer 1011.

The communication controller 1001 controls communication with the imagecapturing apparatus (camera) 320, the operator terminal 210, and thedisplay control apparatus 330 via the network 350. The captured imageacquirer 1002 acquires a captured image from the image capturingapparatus (camera) 320 via the communication controller 1001. The faceimage extractor 1003 extracts a face image from the captured imageacquired by the captured image acquirer 1002. Note that if the imagecapturing apparatus (camera) 320 transmits a face image or the featureof the face image, the face image extractor 1003 is unnecessary. Thecaptured face image database 1004 accumulates the captured face image tobe searchable.

The collation target face image database 1005 accumulates a referenceface image to undergo collation for recognition of the captured faceimage to be searchable. In this example embodiment, the partial regiondatabase 1006 stores a partial region, for which the similarity iscalculated by collation, to be searchable. Note that the partial regiondatabase 1006 may store not only data for a partial region but also analgorithm of generating a partial region.

The partial region generator 1007 generates partial regions of thecaptured face image found from the captured face image database 1004 andthe captured face image found from the collation target face imagedatabase 1005 based on the defined partial region selected from thepartial region database 1006. The partial region similarity calculator1008 associates the partial regions of the captured image and those ofthe collation target face image, which have been generated by thepartial region generator 1007, and compares them, thereby calculatingthe similarity of each pair of partial regions. The face recognizer 1009includes a face recognition table 1091, and recognizes whether the faceimages are similar based on the pattern or the accumulation of thesimilarities of the pairs of partial regions calculated by the partialregion similarity calculator 1008.

The face recognition result and similarity transmitter 1010 transmits,to the display control apparatus 330 via the communication controller1001, the face recognition result indicating whether the faces aresimilar, which has been output from the face recognizer 1009, and thesimilarities of the pairs of partial regions output from the partialregion similarity calculator 1008. Note that the face recognition resultmay be directly sent to the operator terminal 210. The operator'soperation acquirer 1011 receives an operator operation from the operatorterminal 210 via the communication controller 1001, and executesprocessing corresponding to the operator operation.

Note that the face recognition apparatus 340 shown in FIG. 10 performsface recognition based on the captured face image and the collationtarget face image. However, if both the face images are captured faceimages or collation target face images, the face images are read outfrom the same database.

(Captured Face Image Database)

FIG. 11A is a table showing the structure of the captured face imagedatabase 1004 according to this example embodiment. Note that thestructure of the captured face image database 1004 is not limited tothat shown in FIG. 11A.

In association with a captured face image ID 1111 for identifying acaptured face image, the captured face image database 1004 stores animage capturing apparatus ID 1112 for identifying an image capturingapparatus, an image capturing date and time 1113, captured face imagedata 1114, and a feature 1115 of the face image data.

(Collation Target Face Image Database)

FIG. 11B is a table showing the structure of the collation target faceimage database 1005 according to this example embodiment. Note that thestructure of the collation target face image database 1005 is notlimited to that shown in FIG. 11B.

In association with a target face image ID 1121 for identifying a targetface image to be collated, the collation target face image database 1005stores face image attributes 1122 added to the face image, personattributes 1123 added to the person of the face image, face image data1124, and a feature 1125 of the face image data. The face imageattributes 1122 store the acquisition date and time of the face image,the acquisition location of the face image, and the like. The personattributes 1123 store the name, age, sex, address, and the like of theperson.

(Partial Region Database)

FIG. 12 is a table showing the structure of the partial region database1006 according to this example embodiment. Note that the structure ofthe partial region database 1006 is not limited to that shown in FIG.12.

The partial region database 1006 stores a partial region generationalgorithm 1202 in association with a partial region ID 1201 foridentifying a partial region obtained by dividing the face image. Forexample, as the partial region generation algorithm 1202, the size ofmesh division of the face image, that is, H (vertical)×W (horizontal),regions respectively including feature portions such as eyes, a nose,and mouth, a mixture of a feature portion and a mesh (a smaller size ofh (vertical)×w (horizontal)), a mixture of a mesh (a size of H(vertical)×W (horizontal)) and a mesh (a smaller size of h (vertical)×w(horizontal)), and the like are shown.

However, the algorithm is not limited to those shown in FIG. 12. Forexample, a fine region is set as an important region for facerecognition and a coarse region is set as an unimportant region, or viceversa. The partial region generation algorithm 1202 can be changed inaccordance with a method (similarity calculation method) of performingcollating each pair of partial regions. The following partial regiongeneration method is also included. ⋅A region is generated as a mesh bya polygon such as a triangle or hexagon, or a closed curve such as acircle, instead of a rectangle. ⋅Regions in which two or more regionsoverlap each other are generated. ⋅Partial regions are generated basedon an operator operation.

(Face Recognition Table)

FIG. 13 is a table showing the structure of the face recognition table1091 according to this example embodiment. The face recognition table1091 is used to perform face recognition based on the similarities ofthe pairs of partial regions in the face recognizer 1009.

The face recognition table 1091 stores a similarity score 1303 obtainedby assigning a score to the similarity of each pair of partial regionsand a total score 1304 in association with a captured face image ID 1301for identifying a captured face image and a collation target face imageID 1302 for identifying a collation target face image. The facerecognition table 1091 also stores a comparison result 1305 between thetotal score 1304 and a face similarity threshold stored in advance, anda face recognition result 1306 of similarity or dissimilaritycorresponding to the comparison result 1305. Note that the total score1304 may be calculated by giving a region weight to the similarity score1303 of the pair of partial regions. The face similarity threshold ispreferably set in accordance with the partial region division method,the similarity calculation method, and the scoring method.

<<Functional Arrangement of Operator Terminal>>

FIG. 14 is a block diagram showing the functional arrangement of theoperator terminal 210 according to this example embodiment.

The operator terminal 210 includes a communication controller 1401, adisplay image receiver 1402, a display unit 1403, an operation unit1404, and an operator's operation transmitter 1405. If the imagecapturing apparatus (camera) 320 is locally connected to the operatorterminal 210, the operator terminal 210 includes a captured imageacquirer 1406 and a captured image transmitter 1407.

The communication controller 1401 controls communication with the imagecapturing apparatus (camera) 320, the face recognition apparatus 340,and the display control apparatus 330 via the network 350. The displayimage receiver 1402 receives, via the communication controller 1401,normal image data, overlaying image data, image data to be overlaid, orthe like to be displayed on the display unit 1403. For example, acaptured image is received from the image capturing apparatus (camera)320. A notification image of a face recognition result is received fromthe face recognition apparatus 340. Image data obtained by overlaying,on each partial region of the face image, a mark indicating a result ofcomparison with the similarity threshold is received from the displaycontrol apparatus 330. The display unit 1403 displays the image receivedby the display image receiver 1402.

The operation unit 1404 includes an operation processing table 1441, anddetermines operation contents input by the operator from a touch panelor the like. Note that in the case of the touch panel, the operationcontents are also associated with the display contents of the displayunit 1403. The operator's operation transmitter 1405 transmits, via thecommunication controller 1401, the operation contents by the operator tothe respective apparatuses that implement the operation. The capturedimage acquirer 1406 acquires a captured image from the locally connectedimage capturing apparatus (camera) 320. The captured image transmitter1407 transmits the captured image to the face recognition apparatus 340via the communication controller 1401. Note that a communicationterminal incorporating a camera, such as a smartphone or tablet,includes the captured image acquirer 1406 and the captured imagetransmitter 1407.

(Operation Processing Table)

FIG. 15 is a table showing the structure of the operation processingtable 1441 according to this example embodiment. The operationprocessing table 1441 is used to determine an operation input from theoperator to, for example, a touch panel, and convert the operation inputinto an appropriate operation instruction.

The operation processing table 1441 stores operation contents 1503 inassociation with an operation position (coordinates) 1501 on the touchpanel serving as the operation unit 1404 and an operation type 1502 ofthe operator (especially, a finger). Note that practical examples of theoperation action and operation contents by the operator will be omitted.

<<Hardware Arrangement of Display Control Apparatus>>

FIG. 16 is a block diagram showing the hardware arrangement of thedisplay control apparatus 330 according to this example embodiment.

Referring to FIG. 16, a CPU (Central Processing Unit) 1610 is anarithmetic control processor, and implements the functional componentsshown in FIG. 5 by executing a program. A ROM (Read Only Memory) 1620stores permanent data such as initial data and a program, and programs.The communication controller 501 controls communication with the facerecognition apparatus 340, the operator terminal 210, and otherapparatuses via the network 350.

A RAM (Random Access Memory) 1640 is a random access memory used as atemporary storage work area by the CPU 1610. An area to store datanecessary for implementation of the example embodiment is allocated tothe RAM 1640. A similarity 1641 serves as an area to store thesimilarity of each pair of partial regions received from the facerecognition apparatus 340. A threshold 1642 serves as an area to store athreshold to be compared with the similarity 1641. A determinationresult 1643 serves as an area to store a result of comparing thesimilarity 1641 of each pair of partial regions with the threshold 1642.A generated mark 1644 serves as an area to store a mark corresponding tothe determination result 1643, with which the similarity can berecognized at a glance. A face image 1645 serves as an area to store aface image on which the generated mark 1644 is overlaid. The overlaiddisplay data generation table 900 serves as an area to store a table forgenerating overlaid display data, as shown in FIG. 9.Transmission/reception data 1646 serves as an area to store datatransmitted/received via the communication controller 501.

A storage 1650 stores a database, various parameters, or the followingdata or programs necessary for implementation of the example embodiment.As shown in FIG. 6, the face image storage 503 stores a face image inwhich a mark is overlaid on each partial region. As shown in FIG. 7, thethreshold storage 506 stores a threshold to be compared with thesimilarity of each pair of partial regions. As shown in FIG. 8A, themark storage 508 stores a mark to be overlaid, in accordance with aresult of comparing the similarity of each pair of partial regions withthe threshold. The storage 1650 stores the following programs. A displaycontrol apparatus control program 1651 is a program for controlling theoverall display control apparatus 330. A similarity determination module1652 is a module for generating a determination result by comparing thesimilarity of each pair of partial regions with the threshold. A markgeneration module 1653 is a module for generating a mark correspondingto the determination result of the similarity determination module 1652with reference to the mark storage 508. An overlaid image generationmodule 1654 is a module for generating overlaid display data byoverlaying the generated mark 1644 on the face image 1645 andcontrolling display in the operator terminal 210.

Note that programs and data which are associated with thegeneral-purpose functions of the display control apparatus 330 and otherfeasible functions are not shown in the RAM 1640 or the storage 1650 ofFIG. 16.

<<Processing Procedure of Display Control Apparatus>>

FIG. 17 is a flowchart illustrating the processing procedure of thedisplay control apparatus 330 according to this example embodiment. Thisflowchart is executed by the CPU 1610 of FIG. 16 using the RAM 1640,thereby implementing the functional components of FIG. 5.

In step S1701, the display control apparatus 330 receives face images,and stores them in the face image storage 503. In step S1703, thedisplay control apparatus 330 acquires the similarity of each pair ofpartial regions of the face images from the face recognition apparatus340. In step S1705, the display control apparatus 330 executes markgeneration processing of generating a corresponding mark by comparingthe similarity of each pair of partial regions with the threshold.

After generating the mark corresponding to each pair of partial regions,the display control apparatus 330 determines in step S1707 based on anoperator operation in the operator terminal 210 whether the mark is tobe overlaid and displayed. If the operator instructs to display themark, the display control apparatus 330 overlays, in step S1709, themark corresponding to the similarity on the partial region of the faceimage.

In step S1711, the display control apparatus 330 determines whether theall partial regions have been processed. If the all partial regions havenot been processed, the display control apparatus 330 returns to stepS1703 to acquire the similarity of each of the remaining pairs ofpartial regions, and repeats the processes up to step S1709; otherwise,the display control apparatus 330 executes, in step S1713, displaycontrol processing so that the face image on which the marks areoverlaid is displayed on the display unit 1403 of the operator terminal210.

(Mark Generation Processing: Threshold)

FIG. 18A is a flowchart illustrating an example of the procedure of themark generation processing (S1705) according to this example embodiment.FIG. 18A shows an example in which corresponding marks are generatedusing one threshold when the similarity of the pair of partial regionsexceeds the threshold and when the similarity of the pair of partialregions does not exceed the threshold.

In step S1811, the display control apparatus 330 determines whether thesimilarity of each pair of partial regions exceeds the threshold Th0. Ifthe similarity exceeds the threshold Th0, the display control apparatus330 generates, in step S1813, a mark indicating that the partial regionsare similar regions whose similarity exceeds the threshold Th0 so as torepresent the similarity level. On the other hand, if the similaritydoes not exceed the threshold Th0, the display control apparatus 330generates, in step S1815, a mark indicating that the partial regions aredissimilar regions whose similarity does not exceed the threshold Th0 soas to represent the similarity level.

Note that if the mark is overlaid only when the threshold is or is notexceeded, one of steps S1813 and S1815 is necessary. Alternatively, boththe marks may be generated and one of the marks may be selected to beoverlaid at the time of overlaying.

(Mark Generation Processing: First and Second Thresholds)

FIG. 18B is a flowchart illustrating another example of the procedure(S1705) of the mark generation processing according to this exampleembodiment. FIG. 18B shows an example in which corresponding marks aregenerated using the two thresholds when the similarity of the pair ofpartial regions exceeds the higher threshold and when the similarity ofthe pair of partial regions is smaller than the lower threshold.

In step S1821, the display control apparatus 330 determines whether thesimilarity of each pair of partial regions exceeds the threshold Th1(>Th0). If the similarity exceeds the threshold Th1, the display controlapparatus 330 generates, in step S1823, a mark indicating that thepartial regions are similar regions whose similarity exceeds thethreshold Th1 so as to represent the similarity level. On the otherhand, the display control apparatus 330 determines in step S1825 whetherthe similarity of each pair of partial regions is smaller than thethreshold Th2 (<Th0). If the similarity is smaller than the thresholdTh2 (<Th0), the display control apparatus 330 generates, in step S1827,a mark indicating that the partial regions are dissimilar regions whosesimilarity is smaller than the threshold Th2 so as to represent thesimilarity level.

Note that if the mark is overlaid only when the threshold is or is notexceeded, only steps S1821 and S1823 or steps S1825 and S1827 arenecessary. Alternatively both the marks may be generated and one of themarks may be selected to be overlaid at the time of overlaying.

<<Hardware Arrangement of Face Recognition Apparatus>>

FIG. 19 is a block diagram showing the hardware arrangement of the facerecognition apparatus 340 according to this example embodiment.

Referring to FIG. 19, a CPU 1910 is an arithmetic control processor, andimplements the functional components shown in FIG. 5 by executing aprogram. A ROM 1920 stores permanent data such as initial data and aprogram, and programs. The communication controller 1001 controlscommunication with the display control apparatus 330, the operatorterminal 210, and other apparatuses via the network 350.

A RAM 1940 is a random access memory used as a temporary storage workarea by the CPU 1910. An area to store data necessary for implementationof the example embodiment is allocated to the RAM 1940. A captured faceimage and partial region image 1941 serve as areas to store a capturedface image and its partial region images. A collation target face imageand partial region image 1942 serve as areas to store an accumulatedcollation target face image and its partial region images. A partialregion similarity 1943 serves as an area to store a similarity as aresult of collating the corresponding partial regions of the capturedface image and collation target image. The face recognition table 1091serves as an area to store a table for performing face recognition basedon the similarities of the pairs of partial regions, as shown in FIG.13. A face recognition result 1944 serves as an area to store a resultof performing face recognition using the face recognition table 1091.Transmission/reception data 1945 serves as an area to store datatransmitted/received to/from the display control apparatus 330, theoperator terminal 210, and the image capturing apparatus (camera) 320via the communication controller 1001.

A storage 1950 stores a database, various parameters, or the followingdata or programs necessary for implementation of the example embodiment.As shown in FIG. 11B, the collation target face image database 1005 is adatabase for accumulating collation target images. As shown in FIG. 11A,the captured face image database 1004 is a database for storing faceimages captured by the image capturing apparatus (camera) 320. As shownin FIG. 12, the partial region database 1006 is a database for defininga partial region appropriate for collation of the face images. Asimilarity calculation algorithm 1951 stores an algorithm forcalculating the similarity of each pair of partial regions of the faceimages. A face recognition algorithm 1952 stores an algorithm forperforming face recognition based on the similarities of the pairs ofpartial regions. The storage 1950 stores the following programs. A facerecognition apparatus control program 1953 is a program for controllingthe overall face recognition apparatus 340. A partial region collationmodule 1954 is a module for performing comparison and collation togenerate partial regions of the captured face image and collation targetface image and generate the similarity of each pair of partial regions.A face recognition module 1955 is a module for recognizing the faceimage based on the similarities of the pairs of partial regions. A facerecognition result and similarity output module 1956 is a module foroutputting, to the display control apparatus 330 or the operatorterminal 210, the face recognition result as an output from the facerecognition module 1955 and the similarities of the pairs of partialregions as an output from the partial region collation module 1954.

Note that programs and data which are associated with thegeneral-purpose functions of the face recognition apparatus 340 andother feasible functions are not shown in the RAM 1940 or the storage1950 of FIG. 19.

<<Processing Procedure of Face Recognition Apparatus>>

FIG. 20 is a flowchart illustrating the processing procedure of the facerecognition apparatus 340 according to this example embodiment. Thisflowchart is executed by the CPU 1910 of FIG. 19 using the RAM 1940,thereby implementing the functional components of FIG. 10.

In step S2001, the face recognition apparatus 340 receives a capturedface image from the image capturing apparatus (camera) 320.Alternatively, the face recognition apparatus 340 receives a capturedimage and extracts a face image. In step S2003, the face recognitionapparatus 340 acquires a collation target face image from the collationtarget face image database 1005. Note that acquisition of the collationtarget face image may be implemented by sequentially reading out theimages without any algorithm, but it is desirable to read out acollation target face image within a selected range in accordance withthe feature of the captured image or an instruction from the operator,and perform collation.

In step S2005, the face recognition apparatus 340 divides each of thecaptured face image and the collation target face image into identicalpartial regions, and calculates the similarity of each pair of partialregions. In step S2007, the face recognition apparatus 340 performs facerecognition processing based on the similarities of the pairs of partialregions. In step S2009, the face recognition apparatus 340 outputs aface recognition result and the similarities of the pairs of partialregions to the display control apparatus 330 or the operator terminal210.

According to this example embodiment, by controlling to overlay anddisplay, on a face image, marks each indicating whether the similarityexceeds the threshold or not, the operator can grasp a factor for a facerecognition result at a glance, and confirm or modify the facerecognition result on the spot.

Third Example Embodiment

A face recognition system including a display control apparatusaccording to the third example embodiment of the present invention willbe described next. The display control apparatus according to thisexample embodiment is different from that according to the secondexample embodiment in that a new connected region is generated byconnecting adjacent partial regions having similarities falling within apredetermined range, and overlaid and displayed on a face image. Theremaining components and operations are the same as those in the secondexample embodiment. Hence, the same reference numerals denote the samecomponents and operations, and a detailed description thereof will beomitted.

<<Display Screen of Face Recognition Result>>

FIG. 21 is a view showing an outline of the display screen of a terminal210 according to this example embodiment. The display screen shown inFIG. 21 is not limited, and may be the display screen of a mobileterminal, the display screen of a PC, or an apparatus dedicated for facerecognition or surveillance. Note that in FIG. 21, the same referencenumerals as those in FIG. 2A denote the same components and a repetitivedescription thereof will be omitted.

The display screen of the terminal 210 shown at the left end of FIG. 21is the same as that shown on the left side of FIG. 2A. Display screensshown at the center and right end on the upper side of FIG. 21 areobtained by connecting similar regions whose partial region similaritiesexceed a threshold in the display screen at the left end to more readilyrecognize the similar regions. On the other hand, display screens shownat the center and right end on the lower side of FIG. 21 are obtained byconnecting dissimilar regions whose partial region similarities do notexceed the threshold in the display screen at the left end to morereadily recognize the dissimilar regions.

The display screen of the terminal 210 shown at the center on the upperside of FIG. 21 displays regions 2122 to 2124 obtained by connectingregions where marks “◯” 212 of a large size or marks “◯” 213 of a smallsize are adjacent to each other in the display screen at the left end.The display screen of the terminal 210 shown at the right end on theupper side of FIG. 21 displays regions 2132 to 2134 obtained by furtherconnecting the regions 2122 to 2124 in the display screen at the centeron the upper side. On the upper side of FIG. 21, similar regions whosesimilarities exceed the threshold are combined (connected) based on aresult of collating the similarities of the pairs of subdivided partialregions of the face images with the threshold, thereby changing thepartial regions to the similar regions divided more coarsely.

The display screen of the terminal 210 shown at the center on the lowerside of FIG. 21 displays regions 2142 to 2144 obtained by connectingregions where marks “●” are adjacent to each other in the display screenat the left end. The display screen of the terminal 210 shown at theright end on the lower side of FIG. 21 displays regions 2152 to 2154obtained by further connecting the regions 2142 to 2144 in the displayscreen at the center on the lower side. On the lower side of FIG. 21,dissimilar regions are combined (connected) based on a result ofcollating the similarities of the pairs of subdivided partial regions ofthe face images with the threshold, thereby changing the partial regionsto the dissimilar regions divided more coarsely.

Note that the similar regions or dissimilar regions are connected byprocessing of connecting adjacent similar regions or adjacent dissimilarregions, by processing of changing the threshold and connectingremaining portions as similar regions or dissimilar regions, or bychanging the initial size (corresponding to the division number) of thepartial regions. FIG. 21 shows only three stages. However, partialregions of multiple stages can be implemented and the number of stagesat which the operator recognizes a similarity factor more easily whenreferring to the face images can be selected. Since an appropriatenumber of stages is influenced by the size of the display screen, thesize and density of the face image, the shape of the overlaying mark,and the like, the operator may select the number of stages.

<<Functional Arrangement of Display Control Apparatus>>

FIG. 22 is a block diagram showing the functional arrangement of adisplay control apparatus 2230 according to this example embodiment.Note that in FIG. 22, the same reference numerals as those in FIG. 5denote the same functional components and a repetitive descriptionthereof will be omitted.

A partial region connector 2211 includes a region connection table 2212,and connects adjacent similar regions or adjacent dissimilar regionsbased on a comparison result indicating similar regions or dissimilarregions as an output from a comparator 505, and causes a mark generator507 to generate a larger mark indicating a connected region.

Note that as described with reference to FIG. 21, if the initial size(corresponding to the division number) of the partial regions isgradually changed, a face recognition apparatus 340 in the facerecognition system according to this example embodiment changes the size(corresponding to the division number) of the partial regions. However,an apparatus or system in which a display control apparatus 330 and theface recognition apparatus 340 are integrated can uniformly, graduallychange the size (corresponding to the division number) of the partialregions.

(Region Connection Table)

FIG. 23 is a table showing the structure of the region connection table2212 according to this example embodiment. The region connection table2212 is used by the partial region connector 2211 to connect similarregions or dissimilar regions and by the mark generator 507 to generatea connection mark corresponding to a new connected region.

In association with a partial region ID 2301 for identifying a partialregion, the region connection table 2212 stores a similarity condition2302 for connection, a position condition 2303 for connection, and aresult 2304 of whether or not to connect regions. The region connectiontable 2212 also stores connected region coordinates 2305 of theconnected region on the face image after connecting the partial regions,and a connection mark 2306 to be overlaid on the connected region of theface image. Note that as for the connected region coordinates 2305 andthe connection mark 2306, the same coordinates and connection mark areassigned to connected partial regions 2310.

<<Region Connection Processing Procedure>>

FIG. 24 is a flowchart illustrating the procedure of the regionconnection processing of the display control apparatus 2230 according tothis example embodiment. This flowchart is executed by a CPU 1610 ofFIG. 16 using a RAM 1640, thereby implementing the functional componentsof FIG. 22. Note that processing of overlaying the mark of thesimilarity on the face images is similar to the processing procedureshown in FIG. 17 and a description thereof will be omitted.

In step S2401, the display control apparatus 2230 sets a similarityrange to connect partial regions. In step S2403, the display controlapparatus 2230 extracts partial regions within the set similarity range.In step S2405, the display control apparatus 2230 selects adjacentpartial regions from the extracted partial regions within the similarityrange. If there are adjacent partial regions, the display controlapparatus 2230 connects, in step S2407, the adjacent partial regionswithin the set similarity range.

In step S2409, the display control apparatus 2230 determines whether allthe partial regions extracted in step S2403 have been processed. StepsS2405 to S2409 are repeated until all the partial regions are processed.If the processing of all the extracted partial regions ends, the displaycontrol apparatus 2230 generates, in step S2411, a mark corresponding tothe larger connected region generated by connecting the partial regions.In step S2413, the display control apparatus 2230 overlays theconnection mark indicating the connected region of the face image, andcontrols display on the operator terminal 210.

Note that the connection mark may also represent the similarity level bychanging the shape, color, density, transparency, or the like.

According to this example embodiment, since a new connected region isgenerated by connecting adjacent partial regions having similaritiesfalling within the predetermined range, and overlaid and displayed on aface image, the operator can readily grasp a factor for a facerecognition result, and confirm or modify the face recognition result onthe spot.

Fourth Example Embodiment

A face recognition system including a display control apparatusaccording to the fourth example embodiment of the present invention willbe described next. The display control apparatus according to thisexample embodiment is different from those according to the second andthird example embodiments in that partial regions as a factor forerroneous face recognition are generated, and overlaid and displayed onface images. The remaining components and operations are the same asthose in the second and third example embodiments. Hence, the samereference numerals denote the same components and operations, and adetailed description thereof will be omitted.

<<Display Screen of Face Recognition Result>>

FIG. 25 is a view showing an outline of the display screen of a terminal210 according to this example embodiment. The display screen shown inFIG. 25 is not limited, and may be the display screen of a mobileterminal, the display screen of a PC, or an apparatus dedicated for facerecognition or surveillance. Note that in FIG. 25, the same referencenumerals as those in FIG. 2A denote the same components and a repetitivedescription thereof will be omitted.

The display screen of the terminal 210 shown on the upper side of FIG.25 displays a captured face image 2510 and a collation target face image2520 which have been determined by a face recognition apparatus 340 tobe dissimilar, as denoted by reference numerals 2511 and 2521. However,the face images are actually identical except that the person wearsglasses in the collation target face image 2520. If the face recognitionapparatus 340 erroneously recognizes that the face images aredissimilar, the display control apparatus according to this exampleembodiment determines specific partial regions or connected regions as afactor to determine that the face images are dissimilar to causeerroneous recognition, in response to re-recognition by the facerecognition apparatus 340 or indication by the operator from theoperator terminal 210. The display control apparatus according to thisexample embodiment notifies the operator that partial regions orconnected regions 2512 and 2522 cause erroneous recognition.

The display screen of the terminal 210 on the lower side of FIG. 25displays a captured face image 2530 and a collation target face image2540, which have been determined by the face recognition apparatus 340to be similar, as denoted by reference numerals 2531 and 2541. However,the captured face image 2530 includes a male and the collation targetface image 2540 includes a female, and thus the face images are actuallydifferent. If the face recognition apparatus 340 erroneously recognizesthat the face images are similar, the display control apparatusaccording to this example embodiment determines specific partial regionsor connected regions as a factor for determining that the face imagesare similar to cause erroneous recognition, in response tore-recognition by the face recognition apparatus 340 or indication bythe operator from the operator terminal 210. The display controlapparatus according to this example embodiment notifies the operatorthat the similarity of the same pairs of glasses in partial regions orconnected regions 2532 or 2542 causes erroneous recognition.

Note that FIG. 25 shows the rectangular partial regions or connectedregions. However, regions indicating a factor for erroneous recognitionare not limited to the rectangular shape or the number of regions oneach face image is not necessarily one.

<<Face Recognition System>>

FIG. 26 is a sequence chart showing the operation procedure of the facerecognition system according to this example embodiment. Note that inFIG. 26, the same step numbers as those in FIG. 4 denote the same stepsand a repetitive description thereof will be omitted.

After performing overlay and display according to FIG. 4 in steps S401to S419, if the operator notices erroneous recognition by the facerecognition apparatus 340, he/she instructs the operator terminal 210 todisplay a factor for erroneous recognition. In step S2621, the operatorterminal 210 requests a display control apparatus 2630 according to thisexample embodiment to display a factor for erroneous recognition.

If faces are erroneously recognized to be dissimilar, the displaycontrol apparatus 2630 generates, in step S2623, overlaid image data byoverlaying a mark on each of low similar regions. On the other hand, ifthe faces are erroneously recognized to be similar, the display controlapparatus 2630 generates, in step S2625, overlaid image data byoverlaying a mark on each of high similar regions. In step S2627, thedisplay control apparatus 2630 transmits, to the operator terminal 210,overlaid images respectively indicating the regions as an erroneousrecognition factor. In step S2629, the operator terminal 210 notifiesthe operator of the regions as the erroneous recognition factor bydisplaying the overlaid images, as shown in FIG. 25.

<<Functional Arrangement of Display Control Apparatus>>

FIG. 27 is a block diagram showing the functional arrangement of thedisplay control apparatus 2630 according to this example embodiment.Note that in FIG. 27, the same reference numerals as those in FIG. 5 or22 denote the same functional components and a repetitive descriptionthereof will be omitted.

An erroneous recognition region selector 2714 receives an instruction todisplay the regions as the erroneous recognition factor from theoperator terminal 210 and the recognition result of the face recognitionapparatus 340, and selects, if the face images are erroneouslyrecognized to be similar based on a result of comparing the similarityof each pair of partial regions with a threshold by a comparator 505,regions having particularly high similarities. On the other hand, if theface images are erroneously recognized to be dissimilar, the erroneousrecognition region selector 2714 selects regions having particularly lowsimilarities. Then, the erroneous recognition region selector 2714causes a partial region connector 2211 to generate a connected region ofthe selected partial regions.

<<Display Control Processing Procedure>>

FIG. 28 is a flowchart illustrating the procedure of the display controlprocessing of the display control apparatus 2630 according to thisexample embodiment. This flowchart is executed by a CPU 1610 of FIG. 16using a RAM 1640, thereby implementing the functional components of FIG.27. Note that processing of overlaying the mark of the similarity on theface images is similar to the processing procedure shown in FIG. 17 anda description thereof will be omitted.

In step S2801, the display control apparatus 2630 determines whether theoperator terminal 210 has instructed to display the regions as theerroneous recognition factor. If an instruction to display the regionsas the erroneous recognition factor has been sent, the display controlapparatus 2630 acquires the recognition result of the face recognitionapparatus 340 in step S2803.

In step S2805, the display control apparatus 2630 determines whether theacquired face recognition result indicates “similar”. If the facerecognition result indicates “similar”, in step S2807 the displaycontrol apparatus 2630 generates a connected region of partial regionsthat have particularly high similarities and have influenced the facerecognition result indicating “similar”, and generates a mark indicatingthe connected region as the erroneous recognition factor. If the facerecognition result indicates no “similar”, the display control apparatus2630 determines in step S2809 whether the acquired face recognitionresult indicates “dissimilar”. If the face recognition result indicates“dissimilar”, in step S2811 the display control apparatus 2630 generatesa connected region of partial regions that have particularly lowsimilarities and have influenced the face recognition result indicating“dissimilar”, and generates a mark indicating the connected region asthe erroneous recognition factor.

In step S2813, the display control apparatus 2630 overlays the generatedmark on each face image, and controls display of the overlaid images onthe operator terminal 210, as shown in FIG. 25.

According to this example embodiment, since partial regions as a factorfor erroneous face recognition are generated, and overlaid and displayedon face images, the operator can readily grasp a factor for erroneousface recognition, and confirm or modify the face recognition result onthe spot.

Fifth Example Embodiment

A face recognition system including a display control apparatusaccording to the fifth example embodiment of the present invention willbe described next. The display control apparatus according to thisexample embodiment is different from those according to the second tofourth example embodiments in that partial regions as a factor forrecognizing that faces are similar are generated, and overlaid anddisplayed on face images. The remaining components and operations arethe same as those in the second to fourth example embodiments. Hence,the same reference numerals denote the same components and operations,and a detailed description thereof will be omitted.

<<Display Screen of Face Recognition Result>>

FIG. 29 is a view showing an outline of the display screen of a terminal210 according to this example embodiment. The display screen shown inFIG. 29 is not limited, and may be the display screen of a mobileterminal, the display screen of a PC, or an apparatus dedicated for facerecognition or surveillance. Note that in FIG. 29, the same referencenumerals as those in FIG. 2A denote the same components and a repetitivedescription thereof will be omitted.

The display screen of the terminal 210 on the upper side of FIG. 29displays a captured face image 2910 and a collation target face image2920, which have been determined by a face recognition apparatus 340 assimilar faces 2911. The display screen on the upper side of FIG. 29displays face images of a parent and child. Note that both the imagesmay be captured face images or collation target face images. If the facerecognition apparatus 340 recognizes the images as the similar faces2911, when the operator or user inquires about similar portions from theoperator terminal 210 or the mobile terminal, the operator or user isnotified of particularly similar portions by overlaying and displaying,on the face images, partial regions or connected regions 2912 and 2922as a main factor for recognizing the images as the similar faces 2911.

The display screen of the terminal 210 on the lower side of FIG. 29displays a captured face image 2930 and a collation target face image2940, which have been determined by the face recognition apparatus 340as similar faces 2931. The display screen on the lower side of FIG. 29displays a face image of a celebrity on the right side. Note that boththe images may be captured face images or collation target face images.If the face recognition apparatus 340 recognizes the images as thesimilar faces 2931, when the operator or user inquires about similarportions from the operator terminal 210 or the mobile terminal, theoperator or user is notified of particularly similar portions byoverlaying and displaying, on the face images, partial regions orconnected regions 2932 and 2942 as a main factor for recognizing theimages as the similar faces 2931.

Note that FIG. 29 shows the rectangular partial regions or connectedregions. However, regions indicating a factor for the recognition resultindicating similar faces are not limited to the rectangular shape or thenumber of regions on each face image is not necessarily one.

<<Face Recognition System>>

FIG. 30 is a sequence chart showing the operation procedure of the facerecognition system according to this example embodiment. Note that inFIG. 30, the same step numbers as those in FIG. 4 or 26 denote the samesteps and a repetitive description thereof will be omitted.

After performing overlay and display according to FIG. 4 in steps S401to S419, if the operator obtains a recognition result indicating“similar faces” by the face recognition apparatus 340, the operator orthe user instructs the operator terminal 210 or the mobile terminal todisplay a factor for determining “similar faces”. In step S3021, theoperator terminal 210 requests a display control apparatus 3030according to this example embodiment to display a factor for determining“similar faces”.

In step S3025, the display control apparatus 3030 generates overlaidimage data by overlaying a mark on each of high similar regions. In stepS2627, the display control apparatus 3030 transmits, to the operatorterminal 210, overlaid images respectively indicating the regions as afactor for recognizing the “similar faces”. In step S2629, the operatorterminal 210 displays the overlaid images, as shown in FIG. 29, therebynotifying the operator of the regions as the factor for recognizing the“similar faces”. Note that the instruction to display the recognitionfactor from the operator terminal 210 has been explained with referenceto FIG. 30. However, the same applies to processing of a userinstruction from the mobile terminal.

<<Functional Arrangement of Display Control Apparatus>>

FIG. 31 is a block diagram showing the functional arrangement of thedisplay control apparatus 3030 according to this example embodiment.Note that in FIG. 31, the same reference numerals as those in FIG. 5 or22 denote the same functional components and a repetitive descriptionthereof will be omitted.

A similar region selector 3114 receives an instruction to display theregions as a factor for recognizing the “similar faces” from theoperator terminal 210, selects regions having particularly highsimilarities based on an output of a result of comparing the similarityof each pair of partial regions with a threshold by a comparator 505,and causes a partial region connector 2211 to generate a connectedregion of the selected partial regions.

<<Display Control Processing Procedure>>

FIG. 32 is a flowchart illustrating the procedure of the display controlprocessing of the display control apparatus 3030 according to thisexample embodiment. This flowchart is executed by a CPU 1610 of FIG. 16using a RAM 1640, thereby implementing the functional components of FIG.31. Note that processing of overlaying the mark of the similarity on theface images is similar to the processing procedure shown in FIG. 17 anda description thereof will be omitted.

In step S3201, the display control apparatus 3030 determines whether theoperator terminal 210 or the mobile terminal has instructed to displaythe regions as the factor for recognizing the similar faces. If aninstruction to display the regions as the factor for recognizing thesimilar faces has been sent, in step S3203 the display control apparatus3030 generates a connected region of the partial regions that haveparticularly high similarities and have influenced the face recognitionresult indicating “similar”, and generates a mark indicating theconnected region as the factor for recognizing the similar faces. Instep S3205, the display control apparatus 3030 overlays the generatedmark on each face image, and controls display of the overlaid images onthe operator terminal 210, as shown in FIG. 29.

According to this example embodiment, since partial regions as a factorfor recognizing that faces are similar are generated, and overlaid anddisplayed on face images, the operator can readily grasp the factor forrecognizing that the faces are similar.

Sixth Example Embodiment

A face recognition system including a display control apparatusaccording to the sixth example embodiment of the present invention willbe described next. The display control apparatus according to thisexample embodiment is different from those according to the second tofifth example embodiments in that face recognition results areaccumulated and learned, and a region of interest or a region ofnon-interest for performing collation for a generated face image isoverlaid and displayed on the face images. The remaining components andoperations are the same as those in the second to fifth exampleembodiments. Hence, the same reference numerals denote the samecomponents and operations, and a detailed description thereof will beomitted.

<<Display Screen in Face Recognition>>

FIG. 33 is a view showing an outline of the display screen of a terminal210 according to this example embodiment. The display screen shown inFIG. 33 is not limited, and may be the display screen of a mobileterminal, the display screen of a PC, or an apparatus dedicated for facerecognition or surveillance. Note that in FIG. 33, the same referencenumerals as those in FIG. 2A denote the same components and a repetitivedescription thereof will be omitted.

The display screen of the terminal 210 on the upper side of FIG. 33displays a captured face image 3310 and a collation target face image3320 on which partial regions or connected regions 3312 and 3322 as afactor for erroneous recognition are overlaid as regions of non-interest3311 by accumulating a history of recognition results and learning it inthe face recognition apparatus 340. Note that both the images may becaptured face images or collation target face images. Alternatively, oneof the captured face image and the collation target face image may bedisplayed. Based on the accumulation of the history of recognitionresults, for example, even if face images are the same, if the personwears glasses, the eyes and their peripheral regions are erroneouslyrecognized to be dissimilar in the general face images or these faceimages. Thus, marks are overlaid on the face images to exclude the eyesand their peripheral regions from a recognition target. With referenceto the overlaid images, the operator can determine whether the remainingregions are similar or dissimilar without paying attention to the eyesand their peripheral regions, thereby reducing the possibility that theimages are erroneously recognized to be dissimilar.

The display screen of the terminal 210 on the lower side of FIG. 33displays a captured face image 3330 and a collation target face image3340 on which partial regions or connected regions 3332 and 3342 as afactor for erroneous recognition are overlaid as regions 3331 ofinterest by accumulating the history of recognition results and learningit in the face recognition apparatus 340. Note that both the images maybe captured face images or collation target face images. Alternatively,one of the captured face image and the collation target face image maybe displayed. It is leaned based on the accumulation of the history ofrecognition results that, for example, if eyes and noses and theirperipheral regions in general face images or these face images aresimilar, the influence of similarity or dissimilarity of the remainingregions on a recognition result is small. Marks are then overlaid on theface images to exclude the regions other than the eyes and noses andtheir peripheral regions from a recognition target. With reference tothe overlaid images, the operator can avoid erroneous recognition causedby similarity or dissimilarity of the remaining regions, by payingattention to the eyes and noses and their peripheral regions.

Note that FIG. 33 shows the rectangular partial regions or connectedregions. However, regions indicating the regions of non-interest orregions of interest are not limited to the rectangular shape or thenumber of regions on each face image is not necessarily one.

<<Face Recognition System>>

FIG. 34 is a sequence chart showing the operation procedure of the facerecognition system according to this example embodiment. Note that inFIG. 34, the same step numbers as those in FIG. 4 denote the same stepsand a repetitive description thereof will be omitted.

After performing overlay and display according to FIG. 4 in steps S401to S419, a face recognition apparatus 3440 accumulates, in step S3411, aface recognition result and a face image partial region collation result(similarity) in a face recognition history (learning) database to beanalyzable.

The operator inputs, to the operator terminal 210, an instruction tooverlay a region of interest and/or a region of non-interest before facerecognition of himself or herself. Alternatively, a setting may be madein advance in the operator terminal 210. In step S3421, the operatorterminal 210 requests the face recognition apparatus 3440 to overlay anddisplay the region of interest and/or region of non-interest.

In step S3425, the face recognition apparatus 3440 extracts the regionof interest and/or region of non-interest from the informationaccumulated in the learning database. Then, in step S3427, the facerecognition apparatus 3440 transmits the extracted region of interestand/or region of non-interest to a display control apparatus 3430.

The display control apparatus 3430 acquires, in step S3424, the regionof interest and/or region of non-interest from the face recognitionapparatus 3440, and overlays, in step S3431, the region of interestand/or region of non-interest on the captured face image and/orcollation target face image. In step S3433, the display controlapparatus 3430 transmits an overlaid image indicating the region ofinterest and/or region of non-interest to the operator terminal 210. Instep S3435, the operator terminal 210 displays the overlaid image, asshown in FIG. 33, to notify the operator of the region of interestand/or region of non-interest.

Note that FIG. 33 shows the rectangular partial regions or connectedregions. However, regions indicating the factor for the recognitionresult indicating that the faces are similar are not limited to therectangular shape or the number of regions on each face image is notnecessarily one. In addition, if the display control apparatus 3430 andthe face recognition apparatus 3440 are integrally formed, extraction ofthe region of interest and/or region of non-interest and overlaying onthe face images are executed simultaneously.

<<Functional Arrangement of Display Control Apparatus>>

FIG. 35 is a block diagram showing the functional arrangement of thedisplay control apparatus 3430 according to this example embodiment.Note that in FIG. 35, the same reference numerals as those in FIG. 5denote the same functional components and a repetitive descriptionthereof will be omitted.

Under an instruction from the operator terminal 210, aregion-of-interest/region-of-non-interest acquirer 3515 acquires, bycommunication, a mark of the region of interest and/or the region ofnon-interest extracted and transmitted by the face recognition apparatus3440. An image overlay unit 3509 generates overlaid image data byoverlaying the mark of the partial region obtained from the markgenerator 507 or the region of interest and/or the region ofnon-interest acquired by the region-of-interest/region-of-non-interestacquirer 3515 on the face images obtained from a face image storage 503.

Note that the mark may be another mark such as a mark corresponding to aconnected region.

<<Display Control Processing Procedure>>

FIG. 36 is a flowchart illustrating the procedure of the display controlprocessing of the display control apparatus 3430 according to thisexample embodiment. This flowchart is executed by a CPU 1610 of FIG. 16using a RAM 1640, thereby implementing the functional components of FIG.35. Note that processing of overlaying the mark of the similarity on theface images is similar to the processing procedure shown in FIG. 17 anda description thereof will be omitted.

In step S3601, the display control apparatus 3430 determines whether aregion to be overlaid is a region of interest. If a region of interestis to be overlaid and displayed, the display control apparatus 3430acquires, in step S3603, the region of interest from the facerecognition apparatus 3440. If the region to be overlaid is not a regionof interest, the display control apparatus 3430 determines in step S3605whether the region to be overlaid is a region of non-interest. If aregion of non-interest is to be overlaid and displayed, the displaycontrol apparatus 3430 acquires, in step S3607, the region ofnon-interest from the face recognition apparatus 3440. In step S3609,the display control apparatus 3430 overlays the acquired region ofinterest or non-interest on the face images, and controls display on theoperator terminal 210.

Note that an illustration and description of the flowchart when theregion of interest and the region of non-interest are displayed will beomitted.

<<Functional Arrangement of Face Recognition Apparatus>>

FIG. 37 is a block diagram showing the functional arrangement of theface recognition apparatus 3440 according to this example embodiment.Note that in FIG. 37, the same reference numerals as those in FIG. 10denote the same functional components and a repetitive descriptionthereof will be omitted.

A face recognition history (learning) database 3712 accumulates arecognition result based on the similarities of the pairs of partialregions in the face recognition apparatus 3440 and a history ofrecognition target face images and recognition factors. Aregion-of-interest/region-of-non-interest generation transmitter 3713includes a region-of-interest/region-of-non-interest generation table3714, and generates a region of interest/region of non-interest withreference to the face recognition history (learning) database 3712.

(Face Recognition History Database: Learning Database)

FIG. 38 is a table showing the structure of the face recognition history(learning) database 3712 according to this example embodiment. Thestructure of the face recognition history (learning) database 3712 isnot limited to that shown in FIG. 38.

The face recognition history (learning) database 3712 accumulates arecognition result 3803 by the face recognition apparatus and arecognition result 3804 by the operator in association with a capturedimage ID 3801 for identifying a captured face image and a collationtarget face image ID 3802 for identifying a collation target face image.The face recognition history (learning) database 3712 stores learningcontents 3805. The recognition result 3803 by the face recognitionapparatus includes, for example, a recognition result and a regioninfluencing the determination. The recognition result 3804 by theoperator includes, for example, a recognition result and a region ofinterest. The learning contents 3805 include a match or mismatch resultbetween the recognition result by the face recognition apparatus andthat by the operator, a factor for the match, and a factor for themismatch.

Note that although not shown in FIG. 38, recognition conditions such asa face recognition date and time may be accumulated as long as they arenecessary for learning.

(Region-of-Interest and Region-of-Non-Interest Generation Table)

FIG. 39 is a table showing the structure of the region-of-interest andregion-of-non-interest generation table 3714 according to this exampleembodiment. The region-of-interest and region-of-non-interest generationtable 3714 is used by the region-of-interest/region-of-non-interestgeneration transmitter 3713 to generate a region of interest and aregion of non-interest based on the history accumulated in the facerecognition history (learning) database 3712.

The region-of-interest and region-of-non-interest generation table 3714stores partial region coordinates 3902 on the face image and a degree3903 of influence on a recognition result in association with a partialregion ID 3901 for identifying a partial region of the face image. Thedegree 3903 of influence on the recognition result includes a highinfluence score that highly influences the recognition result and a lowinfluence score that does not highly influence the recognition result.The region-of-interest and region-of-non-interest generation table 3714stores data 3904 indicating whether the partial region is set as aregion of interest, and data 3905 indicating whether the partial regionis set as a region of non-interest. Note that in FIG. 39, each of uppertwo partial regions is determined as a region of interest, the thirdregion is determined as a region of non-interest, and the fourth regionis determined as neither a region of interest nor a region ofnon-interest.

Note that FIG. 39 shows only the partial regions. However, the sameapplies to a connected region and the like.

<<Region-of-Interest and/or Region-of-Non-Interest Generation ProcessingProcedure>>

FIG. 40 is a flowchart illustrating the procedure of region-of-interestand/or region-of-non-interest generation processing according to theexample embodiment. This flowchart is executed by a CPU 1910 of FIG. 19using a RAM 1940, thereby implementing the functional components of FIG.37. Note that processing of overlaying the mark of the similarity on theface images is similar to the processing procedure shown in FIG. 17 anda description thereof will be omitted.

In step S4001, the face recognition apparatus 3440 determines whetherthe operator terminal 210 has instructed to generate a region ofinterest and/or region of non-interest. If an instruction to generate aregion of interest and/or region of non-interest has been sent, the facerecognition apparatus 3440 acquires, in step S4003, a captured faceimage from an image capturing apparatus (camera) 320. In step S4005, theface recognition apparatus 3440 acquires a collation target face imageto recognize the acquired captured face image. In step S4007, the facerecognition apparatus 3440 generates a region of interest and a regionof non-interest from the acquired captured face image and collationtarget face image with reference to the history accumulated in the facerecognition history (learning) database 3712. In step S4009, the facerecognition apparatus 3440 transmits the generated region of interestand/or region of non-interest and the face images to the display controlapparatus 3430 to undergo overlay and display.

According to this example embodiment, face recognition results areaccumulated and learned, and a region of interest and a region ofnon-interest for performing collation for a generated face image isoverlaid and displayed on the face images. Thus, the operator canreadily perform face recognition.

Seventh Example Embodiment

A face recognition system including a display control apparatusaccording to the seventh example embodiment of the present inventionwill be described next. The display control apparatus according to thisexample embodiment is different from those according to the second tosixth example embodiments in that it is included in functionalcomponents forming the face recognition system. The remaining componentsand operations are the same as those in the second to sixth exampleembodiments. Hence, the same reference numerals denote the samecomponents and operations, and a detailed description thereof will beomitted.

<<Face Recognition System>>

FIG. 41 is a block diagram showing the arrangement of a face recognitionsystem 4100 according to this example embodiment. The face recognitionsystem 4100 is represented by the minimum number of functionalcomponents forming the face recognition system.

The face recognition system 4100 includes an image capturer 320, a facerecognition apparatus 340, and a display unit 4110. The display unit4110 includes a display control apparatus 330 that generates a markindicating whether the similarity of each pair of partial regionsexceeds a threshold, and overlays the mark on a face image, and anoperator terminal 210 with a display unit 1403 that displays an overlaidimage. The face recognition apparatus 340 includes an accumulator, agenerator that generates partial regions on a captured face image, agenerator that generates partial regions on an accumulated face image, asimilarity calculator that calculates the similarity of each pair ofpartial regions, and a face recognizer based on the similarities of thepairs of partial regions.

According to this example embodiment, the effects of the above exampleembodiments can be implemented by different combinations of variousfunctional components.

Eighth Example Embodiment

A face recognition system including a display control apparatusaccording to the eighth example embodiment of the present invention willbe described next. The display control apparatus according to thisexample embodiment is different from those according to the second toseventh example embodiments in that it is implemented as a displaycontrol server in a cloud server. The remaining components andoperations are the same as those in the second to seventh exampleembodiments. Hence, the same reference numerals denote the samecomponents and operations, and a detailed description thereof will beomitted.

<<Face Recognition System>>

FIG. 42A is a block diagram showing the arrangement of a facerecognition system 4201 according to this example embodiment. Note thatin FIG. 42A, the same reference numerals as those in FIG. 3 denote thesame components and a repetitive description thereof will be omitted.

The face recognition system 4201 includes a cloud server 4260 connectedto a network 350. The cloud server 4260 includes a display controlserver 4230 corresponding to the display control apparatus described inthe above example embodiments and a face recognition server 4240corresponding to the face recognition apparatus.

In accordance with a request to overlay, on a face image, a mark assupport of various face recognition processes, which has been receivedfrom an operator terminal 210 for surveillance arranged in an airport orthe like, and a note PC 21 i or a mobile terminal (smartphone, tablet,or the like) as a user terminal, the cloud server 4260 executesintegration processing to control display of the overlaid image on eachof terminals 210 to 21 n.

FIG. 42B is a block diagram showing the arrangement of a facerecognition system 4202 according to this example embodiment. Note thatin FIG. 42B, the same reference numerals as those in FIG. 3 or 42Adenote the same components and a repetitive description thereof will beomitted.

In the face recognition system 4202, terminals 4211 to 421 n are mobileterminals (smartphones, tablets, or mobile phones) of users. If a videocaptured by a camera incorporated in the mobile terminal of the user istransmitted to a cloud server 4260, the cloud server 4260 displays, onthe screen of the mobile terminal of the user, a mark overlaid image assupport of face recognition processing desired by the user.

According to this example embodiment, the effect of each of the aboveexample embodiments can be implemented by requesting the cloud servereven if the self apparatus has no function.

Ninth Example Embodiment

A face recognition system including a display control apparatusaccording to the ninth example embodiment of the present invention willbe described next. The display control apparatus according to thisexample embodiment is different from those according to the second toeighth example embodiments in that it is included as part of a facerecognition application in a communication terminal serving as a facerecognition system. The remaining components and operations are the sameas those in the second to eighth example embodiments. Hence, the samereference numerals denote the same components and operations, and adetailed description thereof will be omitted.

<<Face Recognition System>>

FIG. 43 is a block diagram showing the functional arrangement of acommunication terminal 4310 serving as a face recognition systemaccording to this example embodiment. Note that FIG. 43 shows newreference numerals “43XX” which denote functional components obtained bycentrally incorporating the functional components shown in FIGS. 5 and10 in the communication terminal 4310. The functional components havingthe same names have the same processing functions, and a repetitivedescription thereof will be omitted.

The communication terminal 4310 includes a communication controller4301, a face recognition application 4302, an image capturer (camera)4303, a display unit 4304, and an operation unit 4305. The facerecognition application 4302 corresponds to an application programdownloaded to the communication terminal 4310. The face recognitionapplication 4302 may be incorporated in the communication terminal 4310in advance.

The face recognition application 4302 includes a collation target faceimage acquirer 4321, a collation target face image database 4322, apartial region generator 4323, and a similarity calculator 4324.Furthermore, the face recognition application 4302 includes a markgenerator 4325 that generates a mark based on a threshold and markinformation, a face recognizer 4326 that recognizes a face image basedon the similarities of pairs of partial regions, and an overlaid imagegenerator 4327 that overlays the mark on a collation target face imageand/or a face image captured by the image capturer (camera) 4303, anddisplays the image on the display unit 4304.

According to this example embodiment, the effect of each of the aboveexample embodiments can be implemented by only the terminal.

Other Example Embodiments

Note that the above example embodiments have explained the example inwhich the display control apparatus and the face recognition apparatusare separately provided. However, the apparatuses may be integrated intoone apparatus or part of one of the apparatuses may be included in theother apparatus. For example, the display control apparatus may have apartial region collation function, and notify the face recognitionapparatus of a collation result. Furthermore, a new apparatus thatcollates partial regions and outputs the result to the display controlapparatus and the face recognition apparatus may be provided as apartial region collation apparatus. That is, this arrangement may beadopted as long as it is possible to implement the functions included inthe display control apparatus and face recognition apparatus accordingto this example embodiment.

While the invention has been particularly shown and described withreference to example embodiments thereof, the invention is not limitedto these example embodiments. It will be understood by those of ordinaryskill in the art that various changes in form and details may be madetherein without departing from the spirit and scope of the presentinvention as defined by the claims.

The present invention is applicable to a system including a plurality ofdevices or a single apparatus. The present invention is also applicableeven when a display control program as an information processing programfor implementing the functions of example embodiments is supplied to thesystem or apparatus directly or from a remote site. Hence, the presentinvention also incorporates the program installed in a computer toimplement the functions of the present invention by the computer, amedium storing the program, and a WWW (World Wide Web) server thatcauses a user to download the program. Especially, the present inventionincorporates at least a non-transitory computer readable medium storinga program that causes a computer to execute processing steps included inthe above-described example embodiments.

1. A display control apparatus comprising: a memory; and a processorcoupled to the memory and configured to: acquire similarity informationbased on a similarity between one or more pairs of partial regions offace images, the similarity being calculated by collation processingbetween the one or more pairs of the partial regions of the face images;generate a display data representing the similarity information based ona factor of similarity indicating whether a partial region is a firstinformation or a second information, the first information being aregion the similarity of which exceeds a threshold and the secondinformation being a region the similarity of which does not exceed thethreshold; and output the display data.
 2. The display control apparatusaccording to claim 1, wherein the processor is further configured toreceive a result of determination whether a first face image and asecond face image are similar, the determination being performed basedon the similarity information, and the processor is configured togenerate the display data representing similarity information indicatingwhether the partial region, the similarity of which is used for thedetermination, is the first information or the second information. 3.The display control apparatus according to claim 1, wherein theprocessor is configured to generate the display data by superimposingthe similarity information on an image representing a human face.
 4. Thedisplay control apparatus according to claim 1, wherein the processor isconfigured to generate the display data by superimposing a same mark ona region corresponding to the second information, and superimposing amark that varies depending on the similarity on a region correspondingto the first information.
 5. A display control method comprising:acquiring similarity information based on a similarity between one ormore pairs of partial regions of face image, the similarity beingcalculated by collation processing between the one or more pairs of thepartial regions of the face images; generating a display datarepresenting the similarity information based on a factor of similarityindicating whether the partial region is a first information or a secondinformation, the first information being a region the similarity ofwhich exceeds a threshold and the second information being a region thesimilarity of which does not exceed the threshold; and outputting thedisplay data.
 6. The display control method according to claim 5,further comprising receiving a result of determination whether a firstface image and a second face image are similar, the determination beingperformed based on the similarity information, and wherein the displaycontrol method comprises generating the display data representingsimilarity information indicating whether the partial region, thesimilarity of which is used for the determination, is the first regionor the second region.
 7. The display control method according to claim5, wherein the display control method comprises generating the displaydata by superimposing the similarity information on an imagerepresenting a human face.
 8. The display control method according toclaim 5, wherein the display control method comprises generating thedisplay data by superimposing a same mark on a region corresponding tothe second information, and superimposing a mark that varies dependingon the similarity on a region corresponding to the first information. 9.A non-transitory computer-readable storage medium storing a program thatcauses a computer to perform: acquiring similarity information based ona similarity between one or more pairs of partial regions of faceimages, the similarity being calculated by collation processing betweenthe one or more pairs of the partial regions of the face images;generating a display data representing the similarity information basedon a factor of similarity indicating whether the partial region is afirst information or a second information, the first information being aregion the similarity of which exceeds a threshold and the secondinformation being a region the similarity of which does not exceed thethreshold; and outputting the display data.
 10. The storage mediumaccording to claim 9, wherein the program further causes the computer toperform receiving a result of determination whether a first face imageand a second face image are similar, the determination being performedbased on the similarity information, and the program causes the computerto perform generating the display data representing similarityinformation indicating whether the partial region, the similarity ofwhich is used for the determination, is the first region or the secondregion.
 11. The storage medium according to claim 9, wherein the programcauses the computer to perform generating the display data bysuperimposing the similarity information on an image representing ahuman face.
 12. The storage medium according to claim 9, wherein theprogram causes the computer to perform generating the display data bysuperimposing a same mark on a region corresponding to the secondinformation, and superimposing a mark that varies depending on thesimilarity on a region corresponding to the first information.
 13. Thedisplay control apparatus according to claim 1, wherein a visualcharacteristic of the display data is determined based on a differencebetween the similarity information and threshold.
 14. The displaycontrol apparatus according to claim 1, wherein size of the display datais determined based on a difference between the similarity informationand threshold.